Long-Term Visitation Value for Deep Exploration in Sparse-Reward Reinforcement Learning

Reinforcement learning with sparse rewards is still an open challenge. Classic methods rely on getting feedback via extrinsic to train the agent, and in situations where this occurs very rarely agent learns slowly or cannot learn at all. Similarly, if receives also that create suboptimal modes of objective function, it will likely prematurely stop exploring. More recent add auxiliary intrinsic encourage exploration. However, lead a non-stationary target for Q-function. In paper, we present novel approach (1) plans exploration actions far into future by using long-term visitation count, (2) decouples exploitation separate function assessing value actions. Contrary existing use models reward dynamics, our off-policy model-free. We further propose new tabular environments benchmarking reinforcement learning. Empirical results classic benchmarks show proposed outperforms rewards, especially presence function. Results suggest scales gracefully size environment.